Bing tells me that while they still support the #! version of crawlable AJAX originally launched by Google, they’re finding it’s not implemented correctly much of the time, and they strongly recommend pushState instead.

Why AJAX Can Be Difficult To Crawl & Index

One common use of AJAX is to make the website experience faster for a visitor, but this implementation can have drawbacks for SEO. Imagine, for instance, a page with several tabs of content.

A web developer could implement this one of several ways.

• A separate URL for each tab – with this implementation, when the visitor clicks a tab, a new request is made to the server for a completely new page. The URL structure might look something like:
  • http://www.example.com/my-web-page?tab=one
  • http://www.example.com/my-web-page?tab=two
  • http://www.example.com/my-web-page?tab=three
• CSS for each tab – with this implementation, the server returns the contents of all tabs with the first page request. When the visitor clicks a tab, the CSS rules cause the browser to hide the contents associated with one tab and show the contents associated with clicked tab. Only one URL is associated with the page, like this:
  • http://www.example.com/my-web-page
• AJAX rending of each tab – with this implementation, when the visitor clicks a tab, only the changing portion of the page is replaced. The URL structure might look something like:
  • http://www.example.com/my-web-page#tab=one
  • http://www.example.com/my-web-page#tab=two
  • http://www.example.com/my-web-page#tab=three

As with most things, pros and cons exist for each option. A separate URL for each tab is easy to share and bookmark and is easy for search engines to crawl and index (they can extract all of the content from each page and have a separate URL to associate with each), but reloading the entire contents of each page can be slow.

CSS for each tab is also easy for search engines to crawl and index, and in some cases, the combined page may rank higher than the same content broken into three pages (due to consolidated incoming links and relevancy signals). But the request for all of that content at once can be the slowest of all to render, and users can’t share or bookmark the page with a secondary tab as active.

AJAX rendering is fastest and enables easy sharing and bookmarking. But, search engines have historically had the hardest time with this implementation. Search engines have trouble extracting content from AJAX/JavaScript calls (although Google has been getting better at it). And # in a URL started as a way to link to content within a page, and so search engines tend to ignore everything in a URL past the #.

Crawlable AJAX

In 2009, Google put together a way to make AJAX crawlable. With this method, a the webpage would use #! rather than #, like this:

• http://www.example.com/my-web-page#!tab=one

For a normal user agent, such as a browser, the # would trigger the AJAX portion of the page, just as it would in a normal AJAX implementation. However, a search engine user agent such as Google would see the #! section of the URL and then request a special version of the page (replacing #! with ?_escaped_fragment_=). In response, the server would return a static version of the page with the contents normally rendered through JavaScript. The benefits of this implementation were that search engines could associate a separate URL with each set of content; and even better, could extract all of that content.

In 2011, Bing began supporting this implementation and included a checkbox in their webmaster tools so site owners could let them know it was being used on a site (they’ve since removed the checkbox, as they’ve gotten better at detecting and crawling it).

As with the other implementations, this has its drawbacks as well, not least of which is the complicated implementation. Bing’s latest blog post notes:

“Developers had to rely on overly-complicated protocols such as “crawlable AJAX”, which uses the #! (“Hash bang”) signature.  This was meant to make AJAX development easier in regards to SEO, but it actually complicated things both for search engines as well as webmasters trying to implement, maintain, and debug their AJAX-driven web pages and applications.

With pushState, we can fully omit the complexity of transforming between “pretty” AJAX URLs and “ugly” static URLs. Search Engines will crawl and index the same URL used by you customers. We are back to business as usual for SEO, including pretty SEO well-understood URLs schema http://domain/path/file?name=value_parameters. This helps you focus on the usual SEO activities (links, page content, etc.) without having to do worry about complex page transformations.”

I asked Bing’s Fabrice Canel, who wrote the post, if Bing still supports the #! version of AJAX URLs, and he told me:

 “We are still supporting the #! crawlable AJAX method but as I said, we do not recommend it at all and we really prefer pushState which is far easier for webmasters and web developers to adopt and maintain.”

HTML5 pushState

With HTML5 pushState, pages can take advantage of the best of both worlds: URLs without # (so search engines can easily index them) and dynamically rendered content for only the change portion of the page (to make things as speedy as possible).

With pushState, URLs look like the first example (a separate URL for each tab), but operate like the third example (AJAX rendering of each tab and the resulting URLs look as follows:

• http://www.example.com/my-web-page?tab=one
• http://www.example.com/my-web-page?tab=two
• http://www.example.com/my-web-page?tab=three

There are other, more complex ways of getting to this same result, such as Hijax, but pushState can be much easier.

Google has been supportive of HTML5 since the beginning (Google’s Maile Ohye in particular began recommending it in conferences since early 2012) and recently published a video in support of HTML5 pushState.

In the video, Google’s Matt Cutts noted:

“A correctly implemented site that uses pushState typically doesn’t need any extra support in order for us to be able to crawl it. We do support both [pushState and #!] and we do handle both standards… but [pushState] is something that I would encourage you to look into and it can be quite helpful in making sure things can be crawlable.”

And now Bing has announced support as well.

Of course, HTML5 has its drawbacks as well, notably that not all older browsers support it, and can require significantly engineering resources to implement (as you’re replacing the current site’s HTML implementation).

If your site uses AJAX-based URLs (either the # versions noted above, or versions that don’t change at all when  content changes) and subsequently, the site is not fully crawled and indexed and you’re looking for solutions, HTML5 pushState is definitely worth looking into.

If your site uses the crawlable #! URLs and isn’t having any trouble being indexed, then you can leave things as they are for now. Both Google and Bing continue to support this implementation.

If you’re considering adding AJAX to your site, make sure you think through the implementation carefully and take into account what content search engines can extract and whether the URLs are indexable.